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REQUEST #85 


What do we know about the impact and implementation of Career Academies? 


RESPONSE 


Career Academies have been around for over 30 years. Their original design stressed facilitating 
the school-to-work transition. In the early 1990’s the focus began to shift to preparing 
participating students for both college and the world of work. In its current form, Career 
Academies is an intervention that generally includes the following components: 

1. School-within-a-school, that provides focused interpersonal support for students; 

2. Combination of academic and career-oriented curricula, using a career theme to 
integrate the two; and 

3. Partnerships with local employers to provide students with career development and 
work-based learning opportunities. 

Educators and policymakers considering this approach should read and discuss the research and 
evaluation for what it can offer about the ki nds of impacts that might be possible. It is also 
important to learn what impacts were desired but not realized. Finally, it is important to 
understand as much as possible about the implementation challenges that have been experienced 
by others and the conditions under which this innovation might fail. Information on this 
intervention is provided in two parts below: Information we found on impacts and information 
we found on implementation. We encourage discussions about both of these aspects of use. 


Impact 


One critical question people want to know is: If we implement this, what might be the types of 
outcomes we could expect given past research and evaluation? 

The Career Academies intervention is unique in that it has been subject to one of the few 
longitudinal experimental studies in education. (That is, it involved random assignment of 
students such that they either participated in a Career Academy or did not.) In addition, we found 
one high quality quasi-experimental study completed after 1990. (Quasi-experimental studies 
are those where students are not randomly assigned but a similar group is found for comparison 
purposes.) 

Here is a summary of the studies that examined impact by comparing two groups of students 
(those who experienced Career Academies and those who did not): 
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■ MDRC (Kemple & Snipes, 2000; Kemple, 2001; Kemple, 2004) conducted a 10-year 
longitudinal experimental study of Career Academies that compared the results of 
randomly assigned participating students to a randomly assigned control group of 
non-participating students. This study found the following long-term impacts: 

□ Career Academies produced positive labor market outcomes for young men; there 
was no difference for young women. 

□ Career Academies had no impact on overall educational attainment. 

□ Among students most at risk of dropping out, Career Academies significantly 
improved high school outcomes. The Academies reduced dropout rates, improved 
attendance, and increased academic course-taking. 

■ One quasi-experimental study (Maxwell & Rubin, 2002) found positive outcomes for 
Career Academies, including increased graduation rates and increased attendance at 
post-secondary institutions. 

Table 1. Outcome Results 


Study 

Outcome 

Measure 

Methodology 1 

Percentage 
Increase over 
Comparison 
Group 

Overall 

Findings 

MDRC Study 
(Kemple, 2004) 

Average 

monthly 

earnings 

Experimental — 
student surveys 

+10.3% 

Labor market 
outcomes are 
very positive for 
men; no 
difference for 
women 

Earned high 
school diploma 

Experimental — 
student surveys 

+.8% 

Slightly positive 
impact on on- 
time graduation 
rates but 
differences 
between 
participants and 
non-participants 
disappear by one 
year later. 

Enrolled in post- 
secondary 
education 

Experimental — 
student surveys 

-1.1% 

No difference. 


1 Each research methodology has its benefits and drawbacks. Descriptive studies are good for describing what is 
happening or how frequently things are happening. Case studies are helpful for understanding how things work in a 
particular setting but they are not good for determining how effective a model is. Correlational studies can look at 
relationships between different factors but cannot determine causality. Experimental designs do the best job of 
determining the effectiveness of interventions but cannot explain why things happen; in addition, they only work in 
some situations. Quasi-experimental studies can make some claims of causality but they have to be very carefully 
designed. 
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Study 

Outcome 

Measure 

Methodology 1 

Percentage 
Increase over 
Comparison 
Group 

Overall 

Findings 

Maxwell & 
Rubin (2002) 

High school 
graduation 

Quasi- 

experimental — 
post-high school 
graduation 
student survey 2 

+8.7% 

The “average” 
career academy 
student has an 
8.7 percentage 
point increase in 
the probability 
of graduating 
from high school 

College 

attendance 

Quasi- 

experimental — 
post-high school 
graduation 
student survey 

+11.6% (two- 
year) 

+17.9% (four- 
year) 

The “average” 
career academy 
student has an 
increased 
probability 
of attending a 
two- or four-year 
post- secondary 
institution 


What Works Clearinghouse (WWC) is a federally sponsored project that sorts through rigorous 
studies on interventions, makes judgments about the studies’ strength, and summarizes the 
findings. WWC has done an intervention report on the effectiveness of Career Academies 
specifically as a dropout prevention program. The intervention report includes results from one 
study (MDRC, summarized above). The report (available at www.whatworks.ed.gov) includes 
the following summary: 

The WWC reviewed seven studies on Career Academies. One of these studies met WWC 
evidence standards, and the remaining six studies did not meet WWC evidence screens. 
Based on this one study, the WWC found potentially positive effects on staying in school, 
potentially positive effects on progressing in school, and no discernible effects on 
completing school. The evidence presented in this report is limited and may change as 
new research emerges (p. 3). 

Summary. The research on the impact of Career Academies suggests that if implemented well, 
you might expect some positive impacts on students at-risk for dropping out and some other 
longer term impacts on certain groups of students, particularly labor market outcomes for males. 
It is possible that there may be some increases in your four-year graduation rate. It is likely, 
however, that Career Academies by themselves will not increase student achievement. 


2 A potential limitation of this study is the extremely low survey response rates: 20% for academy students and 11% 
for comparison students. 
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Implementation 


As important as it is to examine research on impact, it is just as important for a team to consider 
what can be learned from studies on implementation of the approach. 

The MDRC evaluations also looked at issues associated with implementation. They focused 
particularly on the degree of interpersonal supports offered to students. In comparing the degree 
to which a Career Academy offered more interpersonal supports than the larger school in which 
it was embedded, the study concluded that poor implementation of this component was worse 
than no implementation at all: 

In sites where the Academies produced particularly dramatic enhancements in the 
interpersonal support that students received from teachers and peers, the Career 
Academies reduced dropout rates and improved school engagement for both high-risk 
and medium-risk subgroups (about 75 percent of the students served). Academies that 
did not enhance these supports actually increased dropout rates and reduced school 
engagement for some students. (Kemple & Snipes, 2000, p. ES-2 — emphasis added). 

The above finding raises a critical issue for potential implementing sites. If there is not a 
commitment to implementing this interpersonal support component well, some students actually 
may be worse off in a Career Academy setting. 


Discussion 


Given the impact and implementation results described above, some questions to consider when 
implementing Career Academies may include: 

■ What are you expecting to accomplish with Career Academies? Ensure that the goals 
you have for Career Academies are realistic given the research. For example, if well 
implemented, you should be able to expect decreased dropout rates. You should likely 
not expect any changes in student test scores. 

■ Are you committed to putting the interpersonal supports in place that appear to be 
necessary for getting positive results? 

■ What do others who have implemented Career Academies well say about the time 
and staff attitudes and skills necessary to implement well? 

■ Are there any students who might be worse off in a Career Academy setting? What 
could be done to prevent this from happening? 

■ What is a realistic time frame for considering whether the approach should be tried 
and how will stakeholders be involved in the discussion? 

■ How could you begin with a pilot to examine student impact before extensive school- 
wide effort is involved? 
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ANNOTATED BIBLIOGRAPHY 


MDRC Career Academies Evaluation 

The reports below describe the results from the longitudinal Career Academies Evaluation. The 
evaluation used an experimental design that took advantage of the fact that more students applied 
to be part of the Academies than there were spots. As a result, one group of students was 
randomly assigned to participate; the remaining students were the control group. A total of 1,764 
students in nine schools entering the Academies over three years participated in the study. 959 
students were in the experimental Academy group and 805 students were in the control non- 
Academy group. 

Kemple, J. J. (2004). Career academies: Impacts on labor market outcomes and educational 
attainment. New York, NY: MDRC. This report presents findings from surveys sent to students 
four years after students’ anticipated graduation dates. Eighty-two to eighty-three percent of 
students in both groups responded to the surveys. The study came to the following conclusions 
regarding program impacts: 

■ “The Career Academies produced positive and sustained impacts on a range of labor 
market outcomes among the young men in the study sample.” (p. ES-3) 

■ “The Career Academies produced substantial increases in employment and earnings 
for students who entered the programs at high or medium risk of dropping out of high 
school.” (p. ES-5) 

■ “Overall, the Career Academies had no impacts (positive or negative) on educational 
attainment, although high school completion rates and postsecondary enrollment and 
attainment rates were higher than national averages.” (p. ES-5) 

■ “The Career Academies modestly reduced enrollments in post- secondary education 
among those who entered the programs at highest risk of dropping out of high school. 
This does not appear to have diminished the Academies’ impact on employment and 
earnings for this subgroup.” (p. ES- 7) 

Kemple, J. J. (2001). Career academies: Impacts on students’ initial transitions to post- 
secondary education and employment. New York, NY: MDRC. This report presents findings 
from a survey administered to the students 14 months after their scheduled graduation. This 
report presents the following conclusions relative to program impacts: 

■ “On average, the Career Academies had little impact on high school graduation rates 
and initial post- secondary outcomes. The relatively high outcome levels achieved by 
Academy students were matched by those for their non- Academy counterparts.” (p. 
ES-3) 

■ “Among students at high risk of dropping out, the Career Academies’ impacts were 
less pronounced during the year after high school than they were during high school.” 
(p. ES-3) 

■ “In general, the findings for students who entered the program at medium or low risk 
of dropping out of high school were consistent with those for the full sample: 
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Academy and non- Academy students did well relative to national samples, but the 
two groups’ education and labor market outcomes were comparable.” (p. ES-6) 

Kemple, J. J., & Snipes, J. C. (2000). Career academies: Impacts on students' engagement 
and performance in high school. New York, NY: MDRC. This report represents the 
conclusion of the first phase of the evaluation, which looked at students’ results while in school. 
The study used the following data sources: students’ transcripts, student surveys, standardized 
test scores in mathematics and reading, and qualitative data documenting the implementation of 
the program. The report presents the following conclusions: 

■ “The Career Academies in this study increased both the level of interpersonal support 
students experienced during high school and their participation in career awareness 
and work-based learning activities. 

■ The Career Academies substantially improved high school outcomes among students 
at high risk of dropping out. For this group, the Academies reduced dropout rates, 
improved attendance, increased academic course-taking, and increased the likelihood 
of earning enough credits to graduate on time. 

■ Among students least likely to drop out of high school, the Career Academies 
increased the likelihood of graduating on time. The Academies also increased 
vocational course-taking for these students without reducing their likelihood of 
completing a basic core academic curriculum. 

■ In sites where the Academies produced particularly dramatic enhancements in the 
interpersonal support that students received from teachers and peers, the Career 
Academies reduced dropout rates and improved school engagement for both high-risk 
and medium-risk subgroups (about 75 percent of the students served). Academies that 
did not enhance these supports actually increased dropout rates and reduced school 
engagement for some students. 

■ The Career Academies did not improve standardized math and reading achievement 
test scores. 

■ When the findings are averaged across the diverse groups of students in the full study 
sample, it appears that the Career Academies produced only slight reductions in 
dropout rates and modest increases in other measures of school engagement. These 
aggregated findings, however, mask the high degree of variation in effectiveness 
among different groups of students and across the different program contexts.” (pp. 

ES 2-3) 

All MDRC reports are available online at: http://www.mdrc.org/proiect publications 29 l.html 

Maxwell, N. L., & Rubin, V. (2002). High school career academies and postsecondary 
outcomes. Economics of Education Review. 21(2), 137-52. This article examines student 
outcomes associated with career academies. It uses both a detailed data set that contains post- 
secondary information for general, academic, vocational and career academy students from a 
single district and a national (across-district) database of urban, public school students in the 
National Education Longitudinal Study to compare the outcomes from career academy programs 
with those from more traditional programs. It concludes that: 
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■ Students from career academies have a greater probability of graduating from high 
school. 

■ Students from career academy programs are more likely to believe that their high 
school provided them with more education and workplace skills as compared with 
students from the general program. 

■ Students from career academies attend two and four-year colleges at the same level as 
students describing themselves as having had an academic track program. Because 
career academies often draw students from backgrounds that typically define them as 
“at-risk” of school failure, these gains are realized for individuals who are often 
considered less likely to achieve post-secondary success. 

■ Career academies, however, may not be equally effective for all students, and they 
may be expensive to implement. “Females, African-Americans and (native) English 
speakers from the career academies are more likely to have increased educational 
probabilities over comparable students who define themselves as from general and 
vocational programs. The career academy program may have less positive outcomes 
for males, Latinos and non-native English speakers (usually Asian)” (p. 147). 
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